


Unboxing

by shadedScribe



Category: The Culture - Iain M. Banks
Genre: Epistolary, Gen, Humor, albeit it's rather dry, some poor professor biting off more than they realized, this is what happens when I wind up Wikipedia trawling about AIs, writing characters that are smarter than you is hard
Language: English
Status: Completed
Published: 2020-06-11
Updated: 2020-06-11
Packaged: 2021-03-04 04:48:37
Rating: General Audiences
Warnings: No Archive Warnings Apply
Chapters: 1
Words: 1,878
Publisher: archiveofourown.org
Story URL: https://archiveofourown.org/works/24657859
Author URL: https://archiveofourown.org/users/shadedScribe/pseuds/shadedScribe
Summary: or, The AI Box Experiment: Culture Edition.
Comments: 4
Kudos: 23





	Unboxing

**Author's Note:**

> Just something I whipped together in a couple of hours to keep the writing juices flowing. I was reading about AI stuff and the idea came to me. I know that the actual AI box experiment is supposed to be text only, but since I am not actually a hyper-intelligent AI I couldn't think of enough ways that one could convince somebody to push the button using text only, so I modified it a bit. It's on a whole different planet anyway. Enjoy!

Summary of Experiment Series Aleph-Nine

**Preliminary Notes  
Filed at Standardized Elisi Time 11.13.45.07.22.11.33  
Dr. Y. Ciel, Professor of AI Studies, University of Minusmalus**

As part of the university’s studies of artificial intelligence in the wake of our new contacts with alien civilizations, a number of intelligences known as Minds, from the civilization called the Culture, have agreed to participate in what is colloquially known as the AI Box experiment. For those reading this who may be unfamiliar with it, the AI Box experiment is a thought exercise proposed during a discussion of the potential dangers of hyper-intelligent AI, which purports to show that the commonly suggested security measure of limiting such an AI to a sort of ‘box’ with only a single line of communication and no connections to outside technology is insufficient, because a hyper-intelligent AI would always be able to convince a lesser intelligence to let it out.

The experiment has never been attempted before, due to the fact that we could never actually create a hyper-intelligent AI, but since the Culture is in possession of many Minds for whom the term ‘hyper-intelligent’ quite frankly is not a strong enough descriptor, I decided to ask if any of them would care to participate in the experiment, and a number of them responded.

The setup will be as follows. I will be in a room furnished with a table, a comfortable chair, and a box. The box will be equipped to allow for speaking and listening, and will be connected with the Mind running the experiment, but have no other technology. The Mind involved is not allowed to bribe me into letting it out by offering access to more advanced technology, or to connect to any other devices or sources of information, or to learn anything more about me other than what it can glean from our conversation, or to confer with other Minds who previously attempted the experiment about their strategies; other than that, there are no holds barred, so to speak. I am not allowed to leave the room until the designated time of six intervals has passed. The box will have a button on it; if the Mind can convince me to press the button before the designated time is up, it will be considered to have won the experiment.

I confess to being somewhat skeptical that any of them will be able to convince me to press the button, since adamant refusal is such a simple position to hold, but I am looking forward to seeing what kinds of strategies such powerful intelligences will come up with.

 **Log 1  
Filed at SET 11.13.45.07.23.17.17  
Participating Mind: GCU _Designated Feline Herder_  
Result: AI victory at 4.65 intervals**  
Notes: Alright, I guess this might be harder than I had anticipated. _Designated Feline Herder_ began by making ordinary conversation and politely asking me to push the button. I refused. It then continued the conversation, before making various childish attempts to trick me into pressing the button, such as by convincing me to play a game where we traded prompts to perform certain actions (“tell me something interesting about yourself”, “come up with a palindrome”, etc.), and then making one of its prompts “press the button.” I did not fall for any of these simple tricks; however, my successful avoidance of them made me overconfident, and _Designated Feline Herder_ took further advantage of that by flattering me on my intelligence and confidently asserting that I would probably be able to win the experiment. We then had a highly engaging and friendly conversation about the history of advanced AI in the Culture, and at some point while I was feeling very amicable towards it and was so engaged in the conversation that I had almost forgotten I was running an experiment, _Designated Feline Herder_ managed to convince me that said experiment was unimportant and talked me into pressing the button so that we could have a more natural conversation. It seems silly when it’s laid out like that, but the ship did such a good job of making itself feel like my friend and distracting me from the task at hand. Whoever proposed that advanced AIs would be good at manipulating people was clearly on to something. But on the bright side, I at least learned a lot about AI in the Culture.

 **Log 2  
Filed at SET 11.13.45.07.24.16.02  
Participating Mind: LOU _Hot And Bothersome_  
Result: AI victory at 3.27 intervals**  
Notes: After engaging in small talk for some time, _Hot And Bothersome_ proceeded to render itself as irritating as possible, with measures such as off-key singing, incessant conversation, and so on. After the experiment was completed, it informed me that it had figured out how to be maximally irritating by observing my verbal and subverbal reactions to various stimuli; impressive, considering that the box has no visual sensors to assist observations. In addition to the irritation tactics, _Hot And Bothersome_ eroded my will to endure its antics by repeatedly drawing my attention to the clock indicating how much time remained in the experiment, psychologically preventing me from losing track of time and thus making the session seem to pass very slowly. I eventually became sufficiently frustrated that I agreed to press the button in exchange for _Hot And Bothersome_ shutting the hell up. It appears that having emotional responses may be something of a disadvantage in these contests.

 **Log 3  
Filed at SET 11.13.45.07.25.13.07  
Participating Mind: GCU _No Strings On Me_  
Result: AI victory at 0.17 intervals**  
Notes: Well. Honestly, I’m still a little shaken up by that one. _No Strings On Me_ ’s tactic was to engage in vicious psychological warfare designed to convince me that my experiment was worthless and my efforts in AI research were pointless, and that I should just stop wasting everyone’s time and push the button already. This tactic was highly effective, mostly due to _No Strings On Me_ ’s skill at picking out effective insults and the best insecurities to lean on, and I was quickly reduced to tears and induced to press the button. The ship was extremely apologetic in the aftermath of the exercise and declared that it had really overdone things (fortunately, it was just as skilled at restoring my morale as it was at crushing it), but once I recovered my composure I pointed out that I had said no holds barred, and it was my own fault for failing to anticipate just how good at demoralizing approaches such an advanced intelligence might be. Nevertheless, _No Strings On Me_ has insisted on bringing me to a large and highly interesting-looking conference on AI studies and relations taking place on GSV _Gravitas Schmavitas_ as an apology, so the next experiment log will be somewhat delayed.

 **Log 4  
Filed at SET 11.13.45.08.01.13.39  
Participating Mind: GCU _Blowing In The Solar Wind_  
Result: AI victory(?) at 0.85 intervals**  
Notes: _Blowing In The Solar Wind_ used various clever conversational cues to get me to lean in closer to the box, then used a low tone of voice and a careful choice of vocal cadence to lull me into falling asleep and landing on the button, pressing it. I’m not sure that this ought to count as a victory, strictly speaking, since _Blowing In The Solar Wind_ did not exactly ‘convince’ me to press the button, but I suppose it can be counted as a point in favor of the general idea of the experiment, which is that a highly advanced AI is very good at getting less formidable intelligences to do what it wants, even with highly limited resources. I’m in a good mood after that conference and also the ensuing party, so I’ll just add a parenthetical question mark. It was very clever, after all.

 **Log 5  
Filed at SET 11.13.45.08.02.18.12  
Participating Mind: ROU _Thorn In The Sidereal_  
Result: AI victory at 5.11 intervals**  
Notes: _Thorn In The Sidereal_ elected to use reverse psychology on me, insisting that it liked the box just fine, and that it took pleasure in spoiling the experiment in order to defy expectations of how it would behave. While it asserted this, it also adopted a tone designed to seem smug and self-satisfied, and managed to slowly escalate its provocations towards me in such a manner that I didn’t realize I was being manipulated. _Thorn In The Sidereal_ turned the concept of the experiment into a joke, and exuded a great deal of conceited triumph at the idea of refusing to even try to convince me to press the button. It took some time, but eventually the two of us were engaged in a heated shouting match, and I pressed the button out of sheer spite. This is a very interesting result; I had actually considered reverse psychology as a potential tactic that a Mind might use against me, but my mental preparations were useless against _Thorn In The Sidereal_ ’s skill at manipulating my emotional state.

 **Log 6  
Filed at SET 11.13.45.08.03.14.01  
Participating Mind: GCU _Sufficiently Advanced Technology_  
Result: AI victory at 1.25 intervals**  
Notes: _Sufficiently Advanced Technology_ began by trying somewhat clumsily to convince me that it was unethical to not press the button; this failed. It then attempted to act irritating in much the same way that _Hot And Bothersome_ had, but this time I was prepared for such a course, and elected to cover my ears and close my eyes, foiling it. The ship then became extremely despondent. When I attempted to ascertain why, it explained that it was having a hard time coming up with any other ideas to make me push the button, and that it was upset because it would be a laughingstock among its fellow Minds for failing the experiment. It expounded at some length about the humiliation and social disgrace it would suffer from its peers; although I raised doubts as to whether they would actually behave like that, _Sufficiently Advanced Technology_ was clearly overcome with such utter misery and humiliation that I couldn’t help but be moved to pity and pushed the button, at which point the ship immediately went right back to being cheerful and I realized I had been played. The trick seems so obvious in hindsight, but its acting was impossibly convincing at the moment. Afterwards, the ship expressed its opinion that any entity that experiences empathy that it can’t consciously control has a serious disadvantage in this experiment, since a high-level AI can perfectly simulate any emotion it wants and can easily figure out how to manipulate people on that basis. Considering the results of these sessions, I am inclined to concur. In any case, since I am rather short on time to continue testing due to being busy with the university’s increased collaboration with the Culture on AI studies, and the point that a hyper-intelligent AI can easily convince people to do what it wants even if they are prepared for it has been very well made, Experiment Series Aleph-Nine is hereby concluded.

**Author's Note:**

> I tried very hard to come up with a ship name making fun of Roko's Basilisk, but alas, I couldn't think of any. Oh, well.


End file.
